Optimize aligned TVList with lazy column allocation and runtime null bitmap.#17769
Open
luoluoyuyu wants to merge 2 commits into
Open
Optimize aligned TVList with lazy column allocation and runtime null bitmap.#17769luoluoyuyu wants to merge 2 commits into
luoluoyuyu wants to merge 2 commits into
Conversation
…bitmap. Allocate value arrays on write only, use per-chunk BitMaps for explicit nulls, and treat unmaterialized columns as null. Adapt query iterators, flush encoding, and memtable memory estimation for sparse aligned writes.
Collaborator
( serializedSize() still calls getBinaryByValueIndex() unconditionally for TEXT/BLOB/STRING/OBJECT columns. After this PR, a whole value array can be null to represent an unwritten sparse aligned column. In that case WAL size calculation can hit an NPE before serializeToWAL() gets to its valueArray == null handling. 2. TsFileProcessor.java:817 ( / AlignedWritableMemChunk.java:905 ( For an existing aligned memchunk, when an insert expands the TVList into a new chunk, the memory estimate adds only alignedTvListArrayMemCost(). The extra value-array estimate checks only whether the current last chunk is unallocated. If the previous last chunk already had a value array, but the new chunk will allocate value arrays for this insert, those value arrays are not counted. This can underestimate memtable memory usage and weaken write memory control. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Allocate value arrays on write only, use per-chunk BitMaps for explicit nulls, and treat unmaterialized columns as null. Adapt query iterators, flush encoding, and memtable memory estimation for sparse aligned writes.
This PR has:
for an unfamiliar reader.
for code coverage.
Key changed/added classes (or packages if there are too many classes) in this PR